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Chapter  3  -  ADVANCED  UMV  OPERATOR  INTERFACES 

1.1  Data  Display  Systems 

1.1.1  Head-mounted  Displays 

1.1. 1.1  Description  of  Technology 

A  head-mounted  display  (HMD)  presents  real  video  imagery  or  synthetically  generated  visual  imagery  via  a 
head-mounted  optic  system  with  very  small  displays  attached  to  a  helmet,  visor,  or  set  of  spectacles  (see 
Figure  1).  There  is  a  wide  range  of  technologies  and  approaches  associated  with  today’s  HMD  systems, 
including  the  type  and  quality  of  image  display,  monocular  versus  bi-ocular  design,  the  ability  to  display  color 
images,  the  ability  to  effectively  display  stereoscopic  three  dimensional  (3D)  images,  system  size/weight,  and 
the  ability  to  concurrently  view  the  local  external  environment.  As  this  section  is  intended  primarily  as  a  short 
summary  of  the  HMD  technology  and  its  relevance  to  UMVs,  interested  readers  should  refer  to  [1][2][3]  for 
more  comprehensive  coverage  of  this  area. 


Figure  1.  Examples  of  HMDs 

HMDs  can  support  either  immersive  or  augmented  reality  applications,  depending  upon  the  transparency  of 
the  head-mounted  optics.  Immersive  HMDs  require  the  user  to  view  only  the  image  presented  via  the  HMD 
optic  system,  while  augmented  reality  HMDs  allow  the  user  to  “see-through”  the  HMD  display,  thus 
combining  imagery  from  the  HMD  with  the  surrounding  real-world  visual  field.  This  section  only  considers 
immersive  HMD  display  technology,  i.e.,  designs  that  occlude  the  subject’s  view  of  his/her  immediately 
surrounding  physical  environment.  Augmented  reality  technology  is  discussed  in  section - . 

This  section  also  only  considers  head-coupled  HMD  applications  (i.e.,  the  HMD  visual  image  is  updated  in 
response  to  head  movements,  via  a  position  tracking  sensor  that  provides  the  computer  with  the  current  head 
location/orientation  information).  Head-coupled  HMDs  enable  a  synthetically  generated  visual  scene  to  be 
continually  modified  in  response  to  head  movements  so  that,  no  matter  how  the  user  moves,  objects  in  the 
viewed  scene  appear  to  remain  in  stable  locations  (thus  providing  the  impression  that  the  user  is  moving 
within  the  virtually  generated  space).  Alternatively,  in  teleoperated  robotic  systems,  the  motion  of  the  user’s 
head  can  be  used  to  control  the  position/orientation  of  a  remote  camera  (or  other  robotic  action)  [4],  Head- 
coupled  HMDs  offer  the  highest  potential  degree  of  immersiveness  and  maximum  utility.  However  there 
remain  many  limitations  due  to  existing  technology,  as  will  be  discussed  below. 

Depending  upon  the  particular  UMV  application,  it  may  be  critical  for  a  HMD  to  convey  accurate  depth 
information  in  an  intuitive  and  accurate  manner.  Although  all  HMDs  can  convey  a  sense  of  depth  and 
distance  using  conventional  two-dimensional  depth  cues  (including  linear  perspective,  interposition,  relative 
size,  texture  gradients,  etc.),  certain  HMDs  also  have  the  potential  to  portray  depth  via  various  stereoscopic 
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techniques.  Stereopsis  can  provide  an  intuitive,  unambiguous  cue  for  depth  and  it  dominates  most  other  depth 
cues.  Dichoptic  presentation  involves  using  two  monitors  to  portray  a  scene,  one  monitor  per  eye,  each  with 
it’s  appropriate  viewpoint  for  that  eye’s  position  in  space  [5],  This  method  utilizes  binocular  fusion  to  yield 
stereopsis.  Electronic  shutter  glasses  use  one  monitor  to  present  a  stereoscopic  image  by  providing  two 
alternating  views  of  a  scene  (corresponding  to  the  viewpoint  disparity  between  the  two  eyes),  synchronized  to 

the  frame  rate,  such  that  one  interleaved  frame  in  each  pair  is  presented  to  each  eye  [5].  Section - contains 

more  information  on  various  3D  display  technologies. 

1.1. 1.2  Actual  or  Potential  Application  to  UMVs 

HMDs  have  been  found  to  enhance  wide-area  search  and  intercept  operations  performed  by  manned  aircraft 
[6].  A  potential  advantage  of  head-coupled  control  versus  manual  control  over  one’s  viewpoint  is  the  addition 
of  ecologically  relevant  proprioceptive  cues  which  provide  motion  information  based  on  vestibular  inputs, 
joint  angles,  muscle  lengths,  and  tendon  tension  during  head  movements.  Head-coupled  HMDs  are  also 
hypothesized  to  reduce  cognitive  processing  demands  in  achieving  new  viewpoints.  Some  studies  have 
suggested  that  head-coupled  configurations  facilitate  awareness  of  areas  already  searched,  thereby  potentially 
reducing  the  re-scanning  of  those  same  areas  [7],  Thus,  HMD  technology  may  benefit  UMV  operators, 
especially  since  reduced  situational  awareness  is  often  a  by-product  of  current  UMV  control  station  designs 
[8] [9],  It  is  theorized  that  a  HMD  could  enhance  the  operator’s  large-area  searches  and  overall  spatial 
orientation  of  the  remote  environment,  while  larger,  high-resolution  fixed  console  displays  could  be  reserved 
for  any  target  fine  discrimination  tasks.  Other  expected  advantages  of  HMDs  include  hands-free  control  and 
intuitive  operation.  However,  studies  investigating  the  benefits  of  HMDs  have  so  far  produced  mixed  results. 
Below  is  a  summary  of  the  recent  research  in  the  area,  categorized  by  type  of  UMV:  UAV,  UGV  and  UUV. 

1.1. 1.2.1  UAVs 

HMDs  have  been  demonstrated  to  have  a  positive  impact  of  certain  UAV  control  tasks.  A  study  [10]  explored 
UAV  operator  control  of  a  remotely-operated  helicopter  using  an  omni-directional  camera  controlled  by  a 
head-coupled  HMD.  The  HMD  system  was  found  to  promote  operator  immersion  in  the  vehicle,  encouraging 
a  feeling  of  presence  as  though  the  operator  was  in  the  vehicle.  The  HMD  also  resulted  iftfaster  and  more 
accurate  completion  times  in  a  simulated  helicopter  control  task,  compared  to  the  alternative  of  attempting  to 
control  the  helicopter  while  viewing  it  directly  from  the  ground.  These  results  support  claims  that  HMDs  can 
provide  increased  situation  awareness.  However,  the  non-HMD  condition  was  somewhat  lacking  in  that  it  did 
not  include  a  fixed-display  out-the-window  view  from  the  helicopter,  so  it  is  unclear  whether  viewpoint 
location  or  head-coupled  HMD  provided  the  observed  improvements.  Another  study  [11]  explored  HMDs  for 
small  UAV  applications.  The  task  involved  piloting  a  small  UAV  past  a  ground  target  and  then  turning 
around  at  various  distances  to  re-acquire  that  same  ground  target.  The  researchers  found  that  the  use  of  a 
head-coupled  HMD  resulted  in  faster  and  more  successful  re-acquisition  of  the  ground  target  than  when  using 
a  conventional  display  of  imagery  from  the  UAV’s  nose-camera.  However  the  horizontal  field-of  view  was 
nearly  twice  as  large  for  the  HMD  as  compared  to  the  conventional  display,  which  may  have  contributed  to 
these  findings.  Additionally,  all  subjects  complained  of  discomfort  when  wearing  the  HMD.  Nonetheless, 
there  does  exist  research  support  for  the  proposition  that  HMDs  can  provide  greater  situation  awareness 
resulting  in  increased  UAV  operator  performance. 

An  early  experiment  at  the  TNO  Research  Institute  was  conducted  to  explore  the  applicability  of  a  head- 
slaved  camera  system  in  UMV  applications  [12],  To  overcome  some  possible  drawbacks  of  HMDs  (e.g. 
weight),  an  HMD  was  compared  with  a  head-slaved  dome  projection  (Figure  2).  To  overcome  the  possible 
drawbacks  of  transmission  delay,  a  method  was  introduced  to  compensate  for  the  spatial  distortions.  This 
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technique,  called  delay  handling,  preserves  the  correct  spatial  relation  between  the  viewing  direction  of  the 
camera  and  the  operator,  by  presenting  incoming  images  in  the  camera  viewing  direction  at  the  moment  the 
images  were  recorded,  and  not  in  the  actual  viewing  direction  of  the  operator.  The  results  showed  that  delay 
handling  is  successful  in  supporting  the  perception  of  correct  spatial  relations.  No  differences  in  task 
performance  were  found  between  the  actual  HMD  and  the  dome  projection. 


Figure  2:  Dome  projection  in  which  the  camera  direction  is  head-coupled,  and  the  operator  receives 
high  quality  proprioceptive  feedback  on  camera  viewing  direction. 

In  e  follow-on  studies  at  TNO,  researchers  compared  operator  performance  with  head-coupled  camera 
control,  and  Head  Mounted  Displays  (HMDs)  with  manual  camera  control  [13,  14].  Subjects  had  to  locate 
targets  as  fast  as  possible.  The  results  showed  that  head-slaved  camera  control  increased  search  speed  but 
enlarged  the  search  path  as  compared  to  manual  (joystick)  control.  An  increased  susceptibility,  during  head- 
slaved  control,  to  mismatches  between  visual  information  and  proprioceptive  information  may  account  for 
these  findings.  Additional  measures  of  head  movements  showed  that  eye-head  coordination  was  altered 
during  head-slaved  camera  control.  Since  in  these  experiments,  proprioceptive  feedback  was  available  in  the 
manual  control  condition  as  well  (the  images  were  presented  on  a  projection  screen  under  the  correct  camera 
viewing  direction),  the  findings  imply  an  additional  advantage  of  head-slaved  control  compared  with  manual 
control  without  proprioceptive  feedback  (as  would  be  the  case  when  using  a  fixed  monitor).  However,  [15] 
found  that  employing  simulated  HMD  images  projected  onto  a  large  screen  resulted  in  higher  UAV  operator 
performance  than  when  they  used  an  actual  HMD,  and  [16]  found  that  use  of  a  conventional  joystick  for  UAV 
control  resulted  in  better  performance  than  the  head-coupled  HMD.  These  latter  results  converge  with  several 
other  studies,  as  detailed  below. 

Two  experiments  were  conducted  at  AFRL  to  evaluate  the  usefulness  of  HMDs  for  UAV  tasks  involving  the 
search  for  ground  targets  (Figure  3)  [17,  18].  The  overall  approach  was  to  compare  the  utility  of  a  manual 
joystick  with  the  associated  stationary  display  monitor  (the  Baseline  Condition)  to  that  of  various  head- 
coupled  HMD  configurations.  Specifically,  gimbal  camera  orientation  (azimuth  and  elevation  angle)  was 
controlled  via  either  a  right-hand  control  stick  or  head-coupled  HMD,  while  camera  zoom  was  always 
controlled  with  the  left-hand  forward/aft  stick.  In  one  study  [17],  the  task  involved  conducting  a  wide  area 
search  followed  by  a  target  identification  task.  The  wide-area  search  was  conducted  by  using  the  baseline 
configuration  (control  stick  and  stationary  display  monitor)  or  a  head-coupled  HMD.  The  target  identification 
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task  was  always  conducted  using  the  higher  resolution  stationary  display  monitor,  as  the  HMD  did  not  have 
the  required  resolution  to  afford  fine  discrimination.  Thus,  in  the  HMD  conditions,  there  was  a  need  to  switch 
between  displays  between  search  and  identification  tasks.  The  results  failed  to  show  any  benefit  for  HMD- 
based  configurations.  Search  time  was  shorter  and  workload  was  lower  with  the  Baseline  Condition  than  any 
of  the  HMD  conditions.  Additionally,  many  subjects  experienced  discomfort  and  simulator  sickness 
symptoms  with  the  HMD  configuration. 


Figure  2.  UAV  Workstation  with  Head-coupled  HMD  and  Stationary  CRT  Camera  Displays. 

A  follow-on  study  [18]  was  conducted  to  specifically  evaluate  the  utility  of  a  head-coupled  HMD  for  the  SO’s 
conduct  of  a  360-degree  large  area  search  for  multiple  ground  targets.  This  study  did  not  include  the 
additional  target  identification  step  that  had  required  a  switch  from  HMD  to  a  stationary  display  in  the 
previous  study.  Six  camera  control/display  configurations  were  evaluated;  two  involved  the  stationary  display 
monitor  (each  with  a  different  rate  gain  joystick)  and  four  involved  a  HMD.  The  four  HMD  configurations 
varied  in  the  degree  to  which  the  camera  moved  with  head  movements.  One  “hybrid”  configuration  was  also 
evaluated  whereby  the  gimballed  camera  orientation  could  be  controlled  with  both  the  head  and  the  joystick 
simultaneously.  Results  indicated  fewer  unique  targets  were  prosecuted  with  the  HMD  than  with  the  fixed 
display  monitor.  Head-coupled  control  also  resulted  in  more  duplicate  target  designations,  higher  rated 
workload,  and  lower  situation  awareness  ratings.  These  results  suggest  that  there  is  no  clear  advantage  for 
head-coupled  HMDs  in  the  performance  of  large-area  search  tasks.  In  fact,  performance  significantly 
decreased  in  some  experimental  manipulations  involving  the  HMD. 

A  similar  set  of  studies  were  conducted  utilizing  a  simulation  of  a  smaller  UAV  [19,  20].  One  study 
compared  a  conventional  display  monitor  to  a  HMD  for  target  search,  discrimination  and  designation  tasks 
[19].  Although  there  were  no  differences  between  display  conditions  for  target  detection  accuracy,  the 
conventional  display  condition  enabled  more  targets  to  be  correctly  identified  from  further  away  and  allowed 
for  more  accurate  cursor  designation  of  those  targets.  Additionally,  subjects  experienced  far  more  discomfort 
(e.g.,  nausea,  disorientation,  eyestrain)  with  the  HMD  condition.  In  a  follow-on  study  [20],  these  researchers 
explored  the  effect  of  including  various  auditory  cues  (mono,  stereo,  3-D  spatialized)  to  the  ground  target 
location  with  the  comparison  of  visual  display  conditions  (conventional,  head-coupled  HMD).  The  results 
confirmed  earlier  findings  that  conventional  displays  resulted  in  significantly  more  precise  target  designations 
and  fewer  reports  of  discomfort.  However,  although  HMD  conditions  yielded  higher  operator  workload 
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ratings  then  conventional  displays  across  all  conditions,  3-D  spatialized  cueing  reduced  HMD  workload  levels 
significantly. 

1.1. 1.2.2  UGVs 

Designers  argue  that  HMDs  have  characteristics  that  potentially  offer  many  advantages  over  conventional 
UGV  operator  control  units  [21].  Advantages  identified  included  the  system’s  light  weight,  decreased  power 
consumption,  daylight  readability,  and  theorized  improvements  in  operator  SA  and  telepresence.  HMDs  have 
also  been  demonstrated  in  UGV  systems.  HMDs  were  found  to  be  beneficial  during  a  demonstration  of  the 
feasibility  of  utilizing  a  dune  buggy  as  a  UGV  travelling  complex  terrain  [22],  Other  researchers  conducted  a 
study  which  found  telepresence,  created  from  use  of  stereo  TV  imagery  displayed  in  an  HMD,  permitted 
operators’  to  drive  UGVs  at  higher  speeds  and  on  steeper  side  slopes  by  providing  an  enhanced  sense  of 
spatial  /  geographic  awareness  [23],  There  has  also  been  implementation  of  HMDs  into  operational  UGV 
systems.  Man-portable  UGVs  are  completing  missions  in  current  military  operations  with  operators  who  wear 
monocular  HMDs  [24].  Soldiers  are  successfully  controlling  the  UGVs  with  a  portable  joystick  and  HMD  to 
explore  cave  complexes  and  suspected  enemy  compounds.  Packbots’  success  in  combat  environments 
demonstrate  HMDs’  increasing  and  promising  role  in  UGV  control  station  design. 

1.1. 1.2. 3  UUVs 

Although  few  studies  have  been  conducted  in  this  area,  the  potential  value  of  HMDs  to  UUV  systems  seems 
promising  from  underwater  operations  to  operator  training  because  of  HMDs’  capability  of  providing  a 
visually  compelling  sense  of  realism  [25] [26].  Other  researchers  have  described  the  potential  importance  of 
providing  UUV  operators  with  meaningful  cues  for  SA,  good  workspace  visibility,  and  vehicle  behaviour 
feedback  for  effective  performance  [27].  The  testbed  they  designed  to  address  underwater  telerobotics 
included  a  head-coupled  HMD  option.  Though  research  specifically  addressing  HMDs’  effectiveness 
compared  to  other  systems  in  operating  UUVs  is  minimal,  the  difficulties  for  operators  controlling  UAVs  and 
UGVs  are  similar  to  those  in  underwater  vehicles,  so  it  is  reasonable  to  assume  that  research  on  HMDs  in 
these  systems  could  transfer  to  UUVs. 

1.1. 1.3  Research  Challenges 

HMDs  have  improved  considerably  since  they  were  first  introduced  into  the  commercial  market.  However, 
there  is  still  much  research  needed  before  they  achieve  widespread  appeal  in  military  and  consumer 
applications.  Research  areas  discussed  below  include  ergonomic  issues,  resolution,  time  latencies,  field  of 
view,  and  the  occurrence  of  motion-sickness  type  symptoms. 

Ergonomic  Issues 

Ergonomic  issues  associated  with  HMDs  can  be  primarily  attributed  to  anthropometric,  biomechanic,  and 
psychomotor  concerns.  Most  HMDs  involve  some  encumbrance  by  the  user,  though  this  varies  with  particular 
equipment  chosen.  Lack  of  fit  is  a  primary  complaint  of  users  [28].  This  includes  inappropriate  fit,  movement 
limitations,  excessive  weight  and/or  size,  improper  distribution  of  the  weight.  HMD  weight  also  has  the 
potential  to  alter  eye-head  coordination.  Suggestions  exist  for  improving  fit  [28],  Newer  displays  are  being 
developed  to  minimize  size  and  weight  such  that  they  can  be  clipped  onto  existing  eye-pieces,  although  other 
tradeoffs  exist  (limited  resolution,  small  field-of-view,  display  placement  within  larger  visual  field,  etc.). 
Furthermore,  certain  HMDs  enable  the  display  system  to  be  removed  or  rotated  out  of  the  way  to  afford 
intermittent  HMD  use  within  a  larger  real-world  work  task.  However  it  is  unknown  which  method  is  most 
preferable  for  various  UMV  applications.  Additional  information  regarding  ergonomic  issues  can  be  found 
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elsewhere  [1][3][4],  Much  research  is  needed  to  improve  the  many  ergonomic  issues  of  HMDS. 

Spatial  Resolution 

Spatial  resolution  is  a  measure  of  the  level  of  detail  available  in  a  visual  display  [2],  However,  it  can  be  a 
misrepresented  parameter  in  HMD  specifications.  Often,  resolution  is  described  in  terms  of  number  of  pixels 
in  a  display.  However,  the  size  of  the  display  and  its  distance  from  the  observer  also  contribute  to  the 
effective  resolution.  Increasing  display  field-of-view  (FOV)  reduces  effective  resolution  by  enlarging  each 
pixel  in  the  same  manner.  Therefore  a  more  effective  manner  in  which  to  specify  resolution  is  in  terms  of 
visual  angle  subtended  per  pixel,  termed  ‘angular  resolution’.  Angular  resolution  is  poor  in  most  current 
HMDs,  far  lower  then  the  resolving  capability  of  the  human  eye  (approximately  1  arcmin  visual  angle  or  less 
[2]).  Thus  researchers  have  found  resolution  to  be  a  limiting  factor  in  HMD  utilization  [29].  An  additional 
confusion  with  assessing  spatial  resolution  of  color  HMDs  is  associated  with  the  pixel-type  used  for 
determining  angular  resolution.  Color  display  pixels  are  often  formed  by  grouping  3  or  4  monocular  pixels  of 
different  wavelengths  (such  as  red,  green,  blue).  Display  manufacturers  often  report  the  number  of  pixels  and 
angular  resolution  based  upon  the  total  number  of  monocular  pixels  available  instead  of  available  color  pixels. 

Research  is  needed  to  better  define  spatial  resolution  requirements  of  HMDs  for  the  range  of  envisioned  UMV 
tasks.  Additionally,  research  is  needed  to  improve  spatial  resolution.  One  promising  technology  in  this  area  is 
the  virtual  retinal  display  [30], 

Time  Latencies 

Time  delays  exist  between  movements  made  by  the  user’s  head  (which  are  tracked  by  some  position-sensing 
device)  and  the  response  of  the  HMD  scene  to  those  movements,  due  to  delays  in  position  tracking  and  image 
generation  [4],  Time  delays  between  head  movements  and  virtual  image  response  result  in  loss  of  visual 
stability  which  can  affect  task  performance  and  generate  a  sensory  rearrangement  between  visual  and 
vestibular  cues  of  motion.  These  sensory  rearrangements  are  believed  to  induce  simulator  sickness  symptoms 
[31,  32,  33],  When  the  additional  time  delay  associated  with  UMV  datalink  communications  are  factored  in, 
the  total  delay  can  be  on  the  order  of  several  seconds.  Additionally,  time  delays  can  affect  user  acceptance 

[7]- 

Specifications  are  needed  for  acceptable  HMD  time  delay  for  various  UMV  applications,  factoring  in  the 
delays  associated  with  communication  with  the  vehicle.  Acceptable  time  delays  for  UUV  operations  may  not 
be  acceptable  for  fast-moving  UAV  systems.  Research  is  also  needed  on  methods  to  ameliorate  the  effects  of 
time  delay,  such  as  through  the  use  of  prediction  techniques  for  head  motion. 

Display  Field-of-View  (DFOV) 

DFOV  is  the  visual  angle  subtended  by  the  display  screen  from  a  given  observer  location  [34],  This 
parameter,  described  in  terms  of  its  horizontal  and  vertical  components,  is  often  desired  to  be  large  to  promote 
a  sense  of  immersiveness  (i.e.,  presence)  and  to  improve  task  performance  through  the  utilization  of 
peripheral  vision  (limited  DFOV  displays  result  in  the  development  of  different  scanning  strategies). 
However,  a  tradeoff  that  occurs  when  one  tries  to  increase  DFOV  using  an  existing  display  is  a  corresponding 
reduction  of  screen  resolution.  Given  a  fixed  display  size,  the  only  way  to  increase  DFOV  is  to  either 
magnify  the  display  using  optics  or  move  the  eye  closer  to  the  screen.  In  either  case,  pixel  size  increases  in 
the  same  proportion  as  screen  size  (since  both  are  fixed  values).  As  pixel  sizes  increase,  display  resolution 
decreases.  Research  is  needed  to  better  understand  DFOV  requirements  for  various  UMV  applications. 
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Additionally,  the  relation  between  DFOV  and  geometric  FOV  (zoomed  in  or  zoomed  out  images)  and  its 
effect  on  UMV  operator  performance  and  comfort  is  needed  [34]. 

Simulator  Sickness/Cyber  Sickness 

Simulator  sickness  (also  termed  cyber  sickness)  is  a  form  of  motion  sickness  that  occurs  as  a  result  of 
experiencing  computer-simulated  visual  environments  [34],  Symptoms  include  nausea,  fatigue,  headache, 
eye-strain,  dizziness,  malaise,  and  blurred  vision.  Besides  the  deleterious  effects  associated  with  simulator 
sickness,  experiencing  these  symptoms  may  result  in  reduced  desire  to  interact  with  the  provoking  system  in 
the  future,  thus  potentially  hampering  overall  mission  effectiveness.  HMD  usage  has  been  strongly  linked 
with  increased  levels  of  simulator  sickness  in  many  studies  including  those  involving  UMVs  [17]  [19]  [20], 
Although  some  guidelines  exist,  more  research  is  needed  to  fully  characterize  and  alleviate  simulator  sickness 
in  UMV-related  HMD  applications. 

Other  Issues 

Other  research  issues  include  the  need  to  better  understand  and  mitigate  workload  associated  with  HMD 
usage.  Due  to  ergonomic  concerns  as  well  as  the  need  to  constantly  move  one’s  head  to  change  one’s 
viewpoint,  workload  and  fatigue  are  real  concerns  associated  with  this  technology  [19]  and  mitigation 
techniques  must  be  explored  [18].  Head  tracking  technology  and  research  is  also  needed  to  define  minimum 
accuracy  requirements  for  various  UMV  systems  and  to  enable  unencumbered  operations  [35].  Display 
brightness  and  contrast  are  also  issues,  especially  for  applications  in  outdoor  environments. 
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